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LONG-TERM  GOALS 

The  Macaulay  Library  is  the  world’s  largest  archive  of  animal  sounds  and  has  been  selected  by  the 
Office  of  Naval  Research  as  a  major  repository  for  the  deposition,  digital  archival,  review,  and 
retrieval  of  the  many  recordings  of  marine  animals  made  over  the  last  half  century.  Archived  marine 
recordings  pose  challenging  retrieval  problems  given  the  typically  long  intervals  of  silence  between 
animal  sounds  and  the  multiplicity  of  species  detectable  in  any  given  recording.  One  goal  of  this 
project  is  to  design  software  that  will  permit  remote  experts  to  annotate  the  content  of  long  recordings 
archived  at  the  Macaulay  Library  through  their  web  browsers.  Annotations  will  permit  subsequent 
searches  of  the  archive  database  to  retrieve  not  only  suitable  recordings,  but  also  those  parts  of  a 
recording  meeting  the  search  criteria.  The  project  also  seeks  to  define  and  extract  a  set  of  acoustic 
features  from  all  archived  marine  recordings  that  can  be  used  in  subsequent  search  and  retrieval  tasks. 
Both  capabilities  will  be  unique  to  this  sound  collection,  and  will  greatly  enhance  the  accessibility  and 
the  utility  of  the  archive  to  scientists,  students,  educators,  military  personnel,  and  the  media. 

OBJECTIVES 

To  achieve  the  annotation  goals,  the  project  must  provide  a)  a  browser-based  software  tool  for 
visualizing  and  playing  back  digitized  sounds  stored  in  the  archive;  b)  mouse-driven  tools  for 
identifying  specific  segments  within  the  visual  image  of  the  sound;  c)  pull  down  menus  that  allow  the 
annotator  to  assign  standardized  metadata  terms  for  annotation  as  well  as  entry  of  custom  notes  to 
specific  segments  within  recordings;  d)  suitable  metadata  structures  for  storage  of  the  annotations  and 
the  relevant  segment  delimitation  points  and  linkages  to  other  relevant  metadata  fields;  e)  search 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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engines  that  support  the  invocation  of  annotation  terms  during  searches  along  with  other  standard 
criteria;  and  f)  retrieval  tools  that  identify  relevant  parts  within  archived  recordings,  mount  the 
recording,  create  a  visual  image,  and  move  to  the  relevant  annotated  segments.  All  of  these  tools  must 
allow  multiple  annotators  to  add  infonnation  to  the  same  recording,  and  search  engines  that  permit 
either  all-annotator  or  specific-annotator  searches.  To  our  knowledge,  no  archive  currently  provides 
suitable  tools  to  meet  these  requirements.  They  thus  must  be  created  from  scratch  by  this  project. 

To  achieve  the  extracted  feature  goal,  we  need  to:  a)  obtain  a  consensus  from  the  marine  acoustics 
community  on  which  common  measures  would  best  facilitate  their  use  of  the  sound  archive;  b)  create 
the  relevant  algorithms  and  test  them;  c)  implement  the  algorithms  so  that  they  can  be  applied  directly 
to  annotated  segments  in  the  archive;  and  d)  provide  suitable  metadata  structures  to  store  the  extracted 
feature  data  and  link  them  to  the  other  fields  pertinent  to  any  recording.  At  some  later  date,  automatic 
batch  signal  detectors  and  segment  delimiters  will  be  developed  that  will  allow  extraction  on  all 
recordings,  whether  yet  annotated  or  not.  However,  that  step  is  not  part  of  the  current  project. 

APPROACH  AND  WORK  PLAN 

The  annotation  and  feature  extraction  software  tools  are  both  complex  tasks  because  they  require 
seamless  integration  of  visualization,  user  input,  database  design  and  manipulation,  search  engine,  and 
web  usability  components.  The  tasks  have  been  broken  into  discrete  modules  and  specifications  written 
for  each  module,  including  definition  of  deliverables  and  due  dates.  The  Macaulay  Library  metadata 
model  required  a  major  reconstruction  and  transfer  into  an  Oracle  relational  enviromnent.  Critical  to 
this  effort  was  the  incorporation  of  existing  or  development  of  new  metadata  standards  for  habitat  type, 
taxonomy  resolution,  and  classification  of  associated  behaviors.  All  software  for  search,  retrieval,  and 
manipulation  is  being  written  in  Java  to  ensure  cross-platfonn  compatibility.  The  visualization  tool, 
(creating  spectrograms  and  waveforms  of  any  archived  sound),  posed  unique  challenges  because  it  had 
to  work  in  a  browser  environment,  handle  both  audio  and  video  archived  files,  guarantee  proper  digital 
rights  protection,  be  instantaneous  in  operation,  allow  for  segment  selection  by  remote  users,  and  meet 
the  high  standards  for  spectrogram  resolution  demanded  by  the  marine  bioacoustics  community.  Our 
solution  has  been  to  build  this  tool  using  Apple  Quicktime,  a  utility  available  on  all  user  platfonns  for 
free  and  providing  all  the  necessary  requirements.  The  feature  extraction  tools  require  input  from  the 
marine  bioacoustics  community  at  all  stages.  This  is  being  handled  by  a)  including  critical  members  of 
that  community  as  partners  in  the  project,  b)  holding  workshops  where  optimal  features  can  be 
discussed,  and  c)  recruiting  testing  and  critique  input  from  a  variety  of  marine  bioacoustics  labs 
throughout  the  United  States  as  algorithms  are  implemented. 

Marc  Dantzker,  Curator  of  Visual  Media  and  Programming  at  the  Macaulay  Library  is  overseeing  the 
management  of  the  software  projects.  Head  programmer  William  Sandner  delegates  JAVA  tasks  and 
works  on  the  critical  parts  of  the  modules.  Jon  Josephs  writes  functional  specificiations  for  each 
module,  and  writes  software  for  specific  ones.  Rafe  Rosen  and  Gui  Iacino  each  work  on  programming 
other  modules  in  the  project.  Erica  Olsen  is  our  web  interface  designer  and  usability  tester.  Tim 
Levatich  handles  all  the  metadata  modeling,  implementation,  and  database  management.  Roger 
Slothower  is  developing  the  mapping  tools.  The  Quicktime  visualization  tool  is  being  developed  by 
Totally  Hip  Technologies,  Inc.,  a  Vancouver,  British  Columbia  company  that  specializes  in  Quicktime 
extensions.  The  Bioacoustics  Research  Program  at  the  Cornell  Lab  of  Ornithology,  including  partners 
Dr.  Chris  Clark  and  Kurt  Fristrup,  are  providing  critical  input  on  the  optimal  design  of  the  visualization 
tools  and  the  feature  extraction  module.  Partner  Dr.  David  Mellinger  has  also  provided  critical  input  on 
feature  selection,  and  will  implement  the  initial  algorithms  in  a  Matlab  environment  so  that  they  can  be 


distributed,  examined,  and  critiqued  by  participating  marine  bioacoustics  laboratories  around  the 
country.  Dr.  Sue  Moore,  another  partner,  will  be  providing  critical  assistance  in  recruiting  suitable 
participant  labs  and  in  feature  selection. 

The  annotation  project  is  well  underway  (see  below),  and  we  expect  to  post  the  initial  version  on  the 
website  during  Spring  2005.  Subsequent  work  the  remainder  of  the  year  will  focus  on  refinement  and 
correction  of  any  problems  in  the  site,  creation  and  implementation  of  interactive  mapping  tools  (both 
for  archival  and  for  search/retrieval  functions),  and  creation  of  the  initial  steps  needed  to  federate  the 
Macaulay  Library  archive  with  the  OBIS/Seamap  geographic  site,  also  supported  by  ONR. 

The  feature  extraction  project  is  still  in  the  stage  of  identifying  candidate  features  for  implementation. 
The  goal  for  the  coming  year  is  to  implement  these  in  Matlab,  distribute  them  to  participating  marine 
acoustics  labs,  and  solicit  feedback  on  design  and  efficacy  of  the  measures.  Once  the  final  set, 
(currently  estimated  at  20  features),  is  ready,  a  server  at  the  Macaulay  Library  will  be  outfitted  to 
perform  extractions  on  all  recordings  as  they  are  annotated.  These  extracted  features  will  then  become 
available  for  searches  along  with  the  annotations  and  other  metadata  fields. 

WORK  COMPLETED 

The  revision  and  structuring  of  the  metadata  model  in  Oracle  to  accommodate  the  relevant  fields  is 
now  complete.  This  includes  access  to  habitat,  taxonomic  resolution,  and  behavior  classification 
standards.  JAVA  routines  for  search  and  viewing  search  results  have  been  written  and  implemented  in 
newly  designed  web  pages  (Fig.  1).  These  pages  are  currently  being  tested  for  usability. 
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Figure  1.  Sample  web  pages  showing  search  options  for  accessing  marine  sound  archive 


The  visualization  tool  has  proved  challenging,  but  is  now  nearing  completion.  The  spectrogram  tool 
functions  extremely  efficiently  with  good  resolution,  and  the  Quicktime  environment  provides  the 
expected  digital  rights  protection  while  being  universally  accessible.  An  example  of  a  browser¬ 
generated  spectrogram  showing  a  spectrogram  of  a  humpback  whale  song  is  provided  in  Fig.  2. 


Figure  2.  Browser /Quicktime  spectrogram  of  humpback  whale  song  notes.  Spectrogram  moves 
during  playback  so  that  dotted  line  indicates  current  section  being  replayed. 

The  waveform  display  is  currently  being  completed  so  that  users  can  see  either  or  both  visualization  of 
a  sound. 

Playback  causes  the  visual  image  (whether  spectrogram  or  waveform  or  both)  to  move  such  that  the 
part  being  played  will  pass  under  a  fixed  dotted  line  in  the  window.  Users  are  able  to  customize 
spectrogram  time  scales,  frequency  scales,  bandwidth,  overlap,  brightness,  contrast,  and  color  scheme. 
They  are  also  able  to  mark  segments  for  annotation  or  download.  While  meant  only  for  browsing 
archived  sounds,  (not  analyzing  them),  the  current  tool  provides  sufficient  information  that  users  can 
choose  intelligently  between  alternative  sounds  for  subsequent  download.  Access  to  the  visualization 
tools  will  be  free  to  all  users  of  the  site. 

The  feature  extraction  project  hosted  one  workshop  in  August  2004  at  which  considerable  progress 
was  made  on  how  to  identify  and  implement  suitable  features.  Tools  have  been  developed  in  Matlab 
for  distribution  and  examination  by  participating  marine  acoustics  laboratories.  An  initial  list  of 
features  for  implementation  and  distribution  is  currently  being  completed  by  Dr.  Kurt  Fristrup  of  the 
Cornell  Bioacoustics  Research  Program. 

RESULTS 

Major  results  from  this  last  year  include  the  creation  of  a  proposed  new  metadata  standard  for  the 
classification  of  animal  behavior  (http://ethodata.comm.nsdl.org/),  an  online  tool  produced  in 
collaboration  between  the  Cornell  Lab  of  Ornithology  and  uBio  (http://www.ubio.org/)  for  resolution 
of  taxonomic  synonymies,  and  development  of  a  browser-compatible  platform-independent  sound 
visualization  tool  that  includes  appropriate  digital  rights  protections. 


IMPACT  AND  APPLICATIONS 


National  Security 

While  the  U.S.  Navy  has  its  own  archives  of  marine  animal  sounds  that  can  be  used  to  discriminate 
between  man-made  devices  and  animals,  the  new  Macaulay  Library  archive  is  likely  to  become  a 
broader  and  more  definitive  source  as  it  grows  and  users  become  more  adept  at  exploiting  the  new 
tools  to  identify  and  retrieve  specific  sounds.  The  military  will  of  course  have  full  access  to  the  entire 
collection. 

Economic  Development 

Both  commercial  and  public  media  frequently  seek  examples  of  sounds  of  newsworthy  species  from 
the  Macaulay  Library.  The  new  tools  developed  here  will  make  such  requests  much  faster  and  effective 
as  the  media  themselves  will  be  able  to  search  and  audition  sample  sounds  remotely  on  the  web,  pay, 
and  download  the  sounds  immediately. 

Quality  of  Life 

Most  marine  animal  conservation  programs  rely  on  bioacoustic  monitoring  to  assess  community 
health,  diversity,  and  activity.  Providing  an  immediately  accessible  voucher  collection  online  will 
greatly  facilitate  all  such  efforts. 

Science  Education  and  Communication 

The  marine  acoustics  research  community  urgently  needs  a  common  repository  of  sounds  of  each 
marine  species  engaged  in  different  activities.  The  Macaulay  Library  has  volunteered  to  build  such  a 
site  and  is  well  on  its  way  with  over  500  hours  of  digitally  archived  recordings  covering  56  species  of 
marine  mammals  and  130  species  of  fish  and  invertebrates. 

TRANSITIONS 
Economic  Development 

The  Macaulay  Library  is  a  major  resource  of  sound  and  video  recordings  for  public  and  commercial 
media,  museums,  zoos  and  aquaria,  producers  of  products  reproducing  animal  sounds,  wildlife 
identification  devices,  CD  and  DVD  nature  productions,  sound  effects  for  the  movie  industry,  etc. 

Quality  of  Life 

Macaulay  Library  provides  sounds  for  web-based  and  kiosk  public  information  sites,  training  tapes  for 
bioacoustic  censusing  by  wildlife  biologists  and  conservation  staff,  workshops  to  train  wildlife 
recordists  and  bioacoustic  censusing  staff,  etc. 

Science  Education  and  Communication 

Macaulay  Library  is  one  of  the  world’s  primary  resources  for  archived  sounds  of  animals  and  is  thus 
widely  used  by  scientists,  teachers,  and  students.  It  is  also  an  active  member  of  the  National  Science 
Digital  Library  program. 

RELATED  PROJECTS 

Macaulay  Library  also  has  a  concurrent  grant  from  the  NSF-funded  National  Science  Digital  Library 
program  to  maintain  an  NSDL  portal  and  web  pages  that  will  maximize  access  and  utilization  of  the 


anitnal  sound  and  video  collections  for  education  at  all  levels.  It  has  another  (smaller)  NSF  grant  to 
fund  the  development  of  an  international  metadata  standard  for  the  field  of  animal  behavior.  Finally, 
ONR  is  supporting  generously  the  archival  of  the  last  half-century’s  accumulated  field  recordings  of 
marine  animals  at  the  Macaulay  Library.  This  has  included  funds  for  both  equipment  acquisition  and 
staff  salaries  for  archivists. 


